Persist OCI manifest content model per image digest by chruffins · Pull Request #455 · kernel/hypeman

chruffins · 2026-08-26T16:26:23Z

summary

This PR does three things

store image layer information locally to images/content/<digest>/manifest.json
make future cleanup and rebuilding possible
improve local image tagging

Stacked stage of the image-storage project. Persists one content model document per image digest at images/content/<digest>/manifest.json:

manifest digest (immutable identity) and media type
resolved platform (os/arch[/variant])
config blob digest with its ordered rootfs.diff_ids
ordered layer descriptors (compressed digest, size, media type) paired with diff ids by position
a blobReferences() accessor listing config + layer digests for future GC

why

Stages 4/5 need the ordered layer list and diff ids to materialize per-layer artifacts and recompose rootfs without re-reading registries; GC needs to know which OCI cache blobs each ready image still references. Extracted from the existing OCI layout cache during the pull's metadata phase — no second blob downloader.

details

extractManifestModel reads manifest + config from the shared layout cache (system/oci-cache), reusing already-downloaded blobs.
Written atomically in finalizeImage beside the shared content; legacy images simply have no model (read returns nil).
Metadata writes now share a single atomic temp-file-then-rename helper (writeJSONAtomic).

flow change

The change moves tag state from persisted per-reference metadata to one persisted pull claim plus in-memory indexes.

pre-existing flow

CreateImage(request):
    resolve the name/tag to a manifest digest
    lock the image manager

    if metadata exists:
        if failed:
            delete the failed image
            continue as a new build

        update the existing reference:
            ready image -> repoint the tag symlink
            pending image -> record a TagClaim and preserve the previous digest
            persist References[reference] and TagClaims

        return the existing image

    create metadata with RequestedTag, TagGeneration, and References
    create a pending tag symlink
    enqueue the pull/build

build:
    pull into the OCI cache
    unpack layers and convert the rootfs
    mark metadata ready and write it
    claimImageTags:
        try the primary requested tag and every TagClaim
        repoint tags whose generation is still current
        collect the image if no tag was claimed

TagImage(source, target):
    resolve the ready source
    promote content when repositories differ
    persist the target in References and ReferenceGenerations
    install the target symlink
    write metadata and roll back the symlink on failure

new flow

CreateImage(request):
    resolve the name/tag to a manifest digest
    lock the image manager

    if metadata exists:
        if failed:
            delete the failed image
            continue as a new build

        if the request has a tag:
            verify credentials for a pending build
            ready image -> repoint the tag symlink
            pending image -> record the tag as waiting for this digest

        return the existing image

    if the request has a tag:
        increment tagGenerations[repository:tag]
        requestedTags[repository:tag] = digest

    create metadata with RequestedTag, PreviousTagDigest, and TagGeneration
    create a pending tag symlink
    enqueue the pull/build

build:
    pull into the shared OCI cache
    extract one bundle containing:
        container metadata
        manifest model
        layer statistics
    unpack layers and convert the rootfs

    finalize:
        install the disk atomically
        write images/content/<digest>/manifest.json
        mark image metadata ready and write it
        claimRequestedTags:
            repoint every tag currently waiting for this digest
            leave newer tag requests untouched
            collect the image if no tag was claimed

TagImage(source, target):
    resolve the ready source and previous target digest
    promote content when repositories differ
    install the target symlink
    write shared image metadata and roll back symlinks on failure
    increment the target generation
    collect replaced content

startup and readiness waits

pre-existing startup:
    recover tag generations from RequestedTag, TagClaims, and ReferenceGenerations
    leave legacy images in their existing layout

pre-existing WaitForReady(tag):
    walk metadata to find the newest request for the tag

new startup:
    recover tagGenerations and requestedTags from metadata, oldest first
    leave legacy images in place for lazy promotion when needed

new WaitForReady(tag):
    read requestedTags[repository:tag]
    wait directly on that digest

The new manifest.json records the config digest, ordered layer digests, diff IDs, platform, and media types. blobReferences() exposes the config and layer digests for later garbage collection without rereading the registry.

validation

go test ./lib/images ./lib/paths -count=1 — new tests cover synthetic two-layer layouts (ordering, digest/diff-id pairing, platform), write/read roundtrip atomicity, missing-model reads, blob references, and an end-to-end import that lands a ready image with a correct model.
Docker Hub-backed pull tests are rate-limited intermittently in this environment; they passed in the unthrottled window earlier in this work.

Note

Low Risk
Additive on-disk metadata on the image finalize path; no auth or API behavior changes, and missing models are handled for legacy images.

Overview
Adds a persisted OCI manifest content model at images/content/<digest>/manifest.json for each image that finishes the pull/build pipeline. The document captures manifest identity, platform, config blob digest with ordered diff_ids, and ordered layer descriptors (compressed digest, size, media type) so later work can rebuild rootfs from per-layer artifacts and GC can see which OCI cache blobs are still referenced via blobReferences().

During pull, extractManifestModel builds this structure from the shared OCI layout cache in the same metadata phase as existing inspection—no extra registry downloads. finalizeImage writes the model atomically next to shared content and overwrites platform with the resolved manifest platform. Images converted before this change simply have no file; readManifestModel returns nil without error.

Metadata JSON writes now share writeJSONAtomic (temp file + rename), including manifest persistence.

^{Reviewed by Cursor Bugbot for commit 752a6d8. Configure here.}

github-actions · 2026-08-26T18:48:25Z

-->

✱ stlc build

✅ go code · compare

Your SDK build was successful.

generate ✅ → bootstrap ✅ → format ✅

116 files generated at cf7a47a (pushed)
go get github.com/kernel/hypeman-go-staging@cf7a47a8fe1d7812e008ec8d55a5496d3b4bd0e6

✅ python code · compare

Your SDK build was successful.

generate ✅ → bootstrap ✅ → format ✅

232 files generated at 309c4c8 (pushed)

✅ typescript code · compare

Your SDK build was successful.

generate ✅ → bootstrap ✅ → format ✅

138 files generated at fb43b25 (pushed)

Diagnostics: ❗ 0 new / 1 total error, 💡 0 new / 5 total note

Level	Code	Message	Targets

Build metadata

Build	`bd_76FzwzCZ-flowery-drop`
Timestamp	`2026-08-31T22:04:44.266Z`
stlc	`8413509`
Spec hash	`1061c067fc80`
Config hash	`659c3687c3f0`

This comment is auto-generated by stlc and is kept up to date as you push.
If you push new commits, re-run this workflow to update this comment.
Last updated: 2026-08-31 22:05:15 UTC

sjmiller609

1. secondary tag stays stale

starting state

# stable currently resolves to digest A
kernel images create app:stable

# registry updates latest and stable to digest B
docker tag app@sha256:B app:latest
docker tag app@sha256:B app:stable
docker push app:latest
docker push app:stable

# start pulling B through latest
kernel images create app:latest

action

kernel images create app:stable

expected: both tags move to B.
actual: only latest moves; stable remains on A.

2. resource labels are ignored

starting state

kernel images create app:latest --tag team=platform
kernel images wait app:latest

action

kernel images create app:latest --tag team=payments

expected: app:latest reports team=payments.
actual: the reused image retains team=platform.

3. readiness returns too early

starting state

# A is ready
kernel images create app@sha256:A
kernel images tag app@sha256:A app:latest

# start B, then a newer C
kernel images create app@sha256:B --as app:latest
kernel images create app@sha256:C --as app:latest

# C subsequently fails; B remains in progress

action

kernel images wait app:latest

expected: wait for B.
actual: immediately succeeds using old ready image A.

4. eager migration potentially risky

If the migration fails for some reason, it's stuck in image not ready until content/A is manually removed.

Consider a no-op migration instead: leave existing images in the legacy layout, use the shared format only for new images, support reads from both layouts, and add lazy migration only when a future feature requires manifest.json.

chruffins · 2026-09-01T21:22:47Z

bugs addressed + found 1 more, re-running CI now

chruffins force-pushed the hypeship/manifest-layer-model branch from 8255d11 to 1f79ed8 Compare August 26, 2026 18:45

chruffins force-pushed the hypeship/manifest-layer-model branch 2 times, most recently from 1d3b38c to e6fdc4c Compare August 26, 2026 18:53

chruffins force-pushed the hypeship/manifest-layer-model branch from 345e8e0 to 07373a3 Compare August 26, 2026 18:55

chruffins force-pushed the hypeship/manifest-layer-model branch from 07373a3 to e8971b4 Compare August 26, 2026 18:58

chruffins force-pushed the hypeship/manifest-layer-model branch from e8971b4 to 4639430 Compare August 26, 2026 19:26

chruffins force-pushed the hypeship/manifest-layer-model branch from 4639430 to 273909b Compare August 26, 2026 19:30

chruffins force-pushed the hypeship/manifest-layer-model branch from 31c4161 to c6dcc2c Compare August 26, 2026 19:47

chruffins force-pushed the hypeship/manifest-layer-model branch from c6dcc2c to cc7944c Compare August 26, 2026 22:22

chruffins force-pushed the hypeship/manifest-layer-model branch 2 times, most recently from b42b87c to 50bee89 Compare August 31, 2026 21:35

chruffins added 2 commits August 31, 2026 21:36

Add local image tagging API

2614973

Fix image tagging review findings

e177d18

chruffins force-pushed the hypeship/manifest-layer-model branch 2 times, most recently from b992e00 to 76c7b91 Compare August 31, 2026 21:56

chore(stlc): seal custom-code tracking files

90f875f

chruffins force-pushed the hypeship/manifest-layer-model branch from 76c7b91 to 5b0a0df Compare August 31, 2026 21:59

Persist OCI manifest content model per image digest

dd879d9

chruffins force-pushed the hypeship/manifest-layer-model branch from 5b0a0df to dd879d9 Compare August 31, 2026 23:37

Base automatically changed from hypeship/image-tag-api to main September 1, 2026 14:39

Simplify image tag management flow

a6e571e

chruffins force-pushed the hypeship/manifest-layer-model branch from b2941b0 to a6e571e Compare September 1, 2026 15:16

Merge main and resolve image conflicts

77cf736

chruffins marked this pull request as ready for review September 1, 2026 15:45

Harden image promotion and tag cleanup

d5195e4

chruffins requested a review from sjmiller609 September 1, 2026 17:46

sjmiller609 requested changes Sep 1, 2026

View reviewed changes

chruffins added 2 commits September 1, 2026 20:40

Fix shared image tag claims and labels

497610c

Preserve tags for recovered image builds

bc4f6a5

chruffins requested a review from sjmiller609 September 1, 2026 21:22

Navigation Menu

Uh oh!

Persist OCI manifest content model per image digest - #455

Persist OCI manifest content model per image digest#455
chruffins wants to merge 9 commits into
mainfrom
hypeship/manifest-layer-model

chruffins commented Aug 26, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 26, 2026 •

edited

Loading

Uh oh!

sjmiller609 left a comment

Uh oh!

chruffins commented Sep 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

chruffins commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

summary

why

details

flow change

pre-existing flow

new flow

startup and readiness waits

validation

Uh oh!

github-actions Bot commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✱ stlc build

Uh oh!

sjmiller609 left a comment

Choose a reason for hiding this comment

1. secondary tag stays stale

2. resource labels are ignored

3. readiness returns too early

4. eager migration potentially risky

Uh oh!

chruffins commented Sep 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

chruffins commented Aug 26, 2026 •

edited

Loading

github-actions Bot commented Aug 26, 2026 •

edited

Loading